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As  requested,  we  evaluated  whether  certain  tasks  were  within  the  mis¬ 
sion  of  the  Office  of  the  Director,  Operational  Test  and  Evaluation 
(dot&e).1  Specifically,  we  examined  the  Director’s  (1)  role  as  Chairman 
of  the  Test  and  Evaluation  Committee  and  (2)  use  of  early  operational 
assessments.  You  expressed  particular  concern  about  whether  opera¬ 
tional  test  and  evaluation  (ot&e)  is  independent  of  development  testing. 


Results  in  Brief 


Until  recently,  the  Director  had  major  responsibilities  chairing  a  com¬ 
mittee  that  planned,  programmed,  and  budgeted  for  development  test 
resources.  As  a  result,  the  legislatively  mandated  independence  and  sep¬ 
aration  that  must  exist  between  development  and  operational  testing 
was  at  risk.  While  this  committee  was  recently  disestablished,  the 
Department  of  Defense  (dod)  is  considering  a  new  structure.  We  believe 
the  Director  should  function  as  an  adviser — not  an  officer  or  member — 
for  any  future  committee  or  council  that  plans,  programs,  and  budgets 
for  development  test  resources. 

Using  early  operational  assessments,  which  help  to  determine  whether 
weapon  systems  are  ready  for  operational  testing,  is  within  dot&e’s  mis¬ 
sion  because  the  assessments  offer  advice  to  the  dod  acquisition  deci¬ 
sionmakers.  Although  these  assessments  rely  primarily  on  development 
and  not  operational  test  data,  they  can  be  useful  in  filling  a  void  when 
actual  operational  test  results  are  not  available.  However,  these  assess¬ 
ments  are  not  substitutes  for  actual  ot&e  and  the  basis  for  developing 
the  assessments  should  be  fully  disclosed  to  avoid  misunderstanding  as 
to  their  nature  and  use.  .- 


Background 


The  Congress,  concerned  that  ot&e  was  not  receiving  sufficient  emphasis 
and  independent  oversight,  directed  the  establishment  of  dot&e 
(P.L.  98-94,  Sept.  24, 1983).  The  statute  designated  the  Director  as  the 
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'In  practice,  DOT&E  is  used  to  mean  both  the  Director  and  the  office  under  the  Director's  responsi¬ 
bility.  To  avoid  confusion,  we  refer  to  the  Director  as  the  Director  and  to  the  office  as  DOT&E. 
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principal  ot&e  official  within  senior  dod  management  to  ensure  that  sys¬ 
tems  being  acquired  are  ready  for  production.  To  ensure  that  dot&e  is 
independent,  the  Congress  legislated  on  several  occasions,  the  organiza¬ 
tional  separation  between  development  and  operational  testing — the 
two  primary  types  of  dod  testing  during  the  acquisition  process.  In  addi¬ 
tion,  the  Director  may  not  be  assigned  any  responsibility  for  develop¬ 
ment  test  and  evaluation  except  to  provide  advice  to  officials 
responsible  for  such  testing. 

The  roles  and  responsibilities  of  the  Director  are  described  in  10  U.S.C. 
section  138.  According  to  the  statute,  the  Director  is  responsible  for 

•  prescribing  policies  and  procedures  for  ot&e, 

•  providing  guidance  to  and  consulting  with  defense  officials  both  gener¬ 
ally  and  with  respect  to  specific  programs, 

•  monitoring  and  reviewing  all  ot&e, 

•  coordinating  joint  ot&e, 

•  analyzing  and  reporting  on  both  the  adequacy  of  the  testing  and  the 
results,  and 

•  reviewing  and  making  recommendations  to  the  Secretary  of  Defense  on 
all  budgetary  and  financial  matters  relating  to  ot&e. 


Development  test  and  evaluation  is  done  throughout  the  acquisition  pro¬ 
cess  to  ensure  the  attainment  of  technical  performance  specifications, 
program  objectives,  and  weapon  system  supportability.  Development 
testing  is  normally  done  by  the  agency  responsible  for  developing  the 
system.  It  uses  such  techniques  as  modeling,  simulation,  prototypes,  or 
full-scale  engineering  development  system  models  to  determine  the 
extent  that  a  system  meets  technical  specifications. 

ot&e  is  the  field  test,  under  realistic  combat  conditions,  of  major  weapon 
systems,  equipment,  components,  or  munitions  for  the  purpose  of  deter¬ 
mining  the  operational  effectiveness  and  suitability2  of  the  weapons, 
equipment,  or  munitions  used  in  combat  by  typical  military  users.  Each 
service  has  an  operational  test  agency  that  is  responsible  for  ot&e, 
reporting  test  results,  and  providing  an  evaluation  of  the  tested  system’s 
operational  effectiveness  and  suitability. 


2Operatfc>nal  effectiveness  means  the  ability  of  a  system  to  accomplish  its  mission  when  placed  in  use 
in  the  planned  operational  environment  Operational  suitability  is  the  degree  to  which  a  system  can 
be  placed  satisfactorily  in  field  use  considering,  among  other  factors,  availability,  maintainability, 
and  logistics  support 
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In  November  1989,  the  Congress  added  provisions  concerning  ot&e  in 
connection  with  the  decision  to  go  beyond  low-rate  initial  production.3 
The  Congress  stated  that  ot&e  does  not  include  an  operational  assess¬ 
ment  based  exclusively  on  (1)  computer  modeling,  (2)  simulation,  or 
(3)  an  analysis  of  system  requirements,  engineering  proposals,  design 
specifications,  or  any  other  information  contained  in  program 
documents. 


The  Director’s  Role  in 
Development  Test 
Resources 
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The  Deputy  Secretary  of  Defense  appointed  the  Director  as  Chairman  of 
the  Test  and  Evaluation  Committee.  As  Committee  Chairman,  the 
Director  presided  over  the  operations  of  the  Committee,  which  was 
responsible  for  planning,  programming,  and  budgeting  for  development 
test  resources.  As  a  result,  the  Director’s  independence  was  jeopardized 
because  the  Director  had  influence  over  the  types  of  development  test 
assets  used  by  the  services.  Responsibility  for  development  test 
resources  rests  with  the  services.  We  believe  the  Director  should  func¬ 
tion  as  an  adviser  on  any  future  committee  or  council  that  is  given 
responsibility  for  development  test  resources. 

In  1987,  the  Congress  inserted  language  in  dot&e’s  statute  through  the 
National  Defense  Authorization  Act  for  Fiscal  Years  1988-89,  which 
states  that  “the  Director  may  not  be  assigned  any  responsibility  for 
developmental  test  and  evaluation,  other  than  the  provision  of  advice  to 
officials  responsible  for  such  testing.”  According  to  the  conference 
report  on  this  act,  the  language  was  inserted  to  ensure  that  responsibili¬ 
ties  for  operational  testing  are  separate  from  functions  associated  with 
development  testing.  The  conference  report  further  stated  that  the  Sec¬ 
retary  of  Defense  should  refrain  from  any  realignment  or  new  arrange¬ 
ment  of  test  and  evaluation  activities,  oversight  responsibilities,  or 
functions. 

We  recently  issued  an  unclassified  version  of  a  classified  report4  that 
criticized  dot&e’s  management  of  the  ot&e  Capability  Improvement  Pro¬ 
gram.  In  that  report,  we  stated  that  dot&e  was  performing  management 
functions  by  acquiring  operational  test  resources  used  in  testing  major 
weapon  systems.  We  recommended  that  the  Secretary  of  Defense  direct 


3  Low-rate  Initial  production  is  the  production  of  a  system  in  the  minimum  quantity  needed  to  conduct 
OT&E,  to  establish  an  initial  production  base  for  the  system,  and  to  permit  an  orderly  increase  in  the 
production  rate  for  the  system  sufficient  to  lead  to  full-rate  production  upon  the  successful  comple¬ 
tion  of  operational  testing. 

4(GAO/NSIAD-90- 141,  Mar.  30, 1990). 
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and  assure  that  the  services  plan,  program,  and  budget  for  adequate  test 
resources  needed  to  conduct  operational  testing  of  weapon  systems 
effectively.  We  explained  that  the  Director  should  continue  to  perform 
oversight  and  policy  functions  by  reviewing  and  making  recommenda¬ 
tions  to  the  Secretary  of  Defense  to  assure  that  adequate  test  resources 
are  acquired. 


The  Director  Influenced 
the  Committee  in 
Development  Test 
Resource  Matters 


The  Test  and  Evaluation  Committee  was  established  to  support  the 
Under  Secretary  of  Defense  for  Acquisition,  who,  by  law,  is  responsible 
for  establishing  policies  for  acquiring  dod’s  weapon  systems  and  the 
development  testing  of  such  systems.  (See  10  U.S.C.  133(bX2).)  The 
Committee  was  to  provide  a  forum  for  key  representatives  from  the 
Office  of  the  Secretary  of  Defense,  the  services,  and  other  agencies  to 
identify  and  resolve  issues  and  to  develop  recommendations  on  the  dod 
acquisition  system  in  the  test  and  evaluation  area. 

The  Committee’s  charter  specified  that  the  Chairman  had  major  roles 
and  responsibilities  regarding  the  operations  of  the  Committee.  The 
Committee  Chair  was  to,  among  other  things,  direct  and  supervise  oper¬ 
ations  of  the  Committee,  schedule  and  preside  at  meetings,  select  items 
submitted  by  the  Committee  members  to  be  addressed  by  the  Com¬ 
mittee,  and  establish  panels  to  carry  out  Committee  assigned  projects. 
The  Committee  attempted  to  obtain  a  consensus  among  all  members 
regarding  matters  brought  before  the  Committee.  As  such,  the  Director 
participated  in  the  decision-making  process  by  concurring  or  not  concur¬ 
ring  on  these  matters,  thus  having  influence  over  the  types  of  test  assets 
used  by  the  services. 

As  Committee  Chairman,  the  Director  initially  provided  advice  and 
presented  recommendations  on  various  test  and  evaluation  issues  to  the 
Deputy  Secretary  of  Defense  for  a  decision.  In  1986,  the  issues  included 
such  things  as  a  proposed  recommendation  to  improve  space  system  test 
capabilities.  In  1988,  the  Director  recommended  establishing  the  Central 
Test  and  Evaluation  Investment  Program,  which  received  $83  million  in 
fiscal  year  1990  for  the  purpose  of  acquiring  primarily  development  test 
resources  for  new  weapon  systems.  The  Director  believed  that  the  lack 
of  development  and  operational  test  investments  by  the  services  was 
reaching  crisis  proportions. 

Instead  of  continuing  to  serve  in  an  advisory  capacity  by  presenting  rec¬ 
ommendations  on  test  resources  to  the  Deputy  Secretary  of  Defense,  the 
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The  Committee  May  Be 
Replaced  by  a  Council 


Director  became  a  manager  of  these  resources  by  taking  on  responsi¬ 
bility  for  the  program.  In  November  1988,  the  Deputy  Secretary  of 
Defense  created  the  program  and  directed  that  it  be  managed  by  the 
Committee.  On  November  30, 1988,  management  of  the  program  was 
established  and  the  Committee  Chairman  assumed  responsibility  and 
accountability  for  the  management  and  effective  allocation  of  the  pro¬ 
gram  funds.6  Management  of  test  resources  may  dilute  dct&e’s  oversight 
function  because  it  gives  dot&e  a  direct  influence  over  the  types  of  test 
assets  acquired  and  used  by  the  services.  For  example,  dot&e  may  be 
reluctant  to  criticize  a  test  plan  that  contains  inadequate  testing 
resources  if  that  office  was  responsible  for  managing  or  played  a  part  in 
the  acquisition  of  such  resources. 

Once  the  Committee  approved  the  projects  to  be  funded  under  the  pro¬ 
gram,  the  Committee’s  resource  panel  established  the  parameters  for  the 
projects  that  were  to  be  executed  by  the  lead  services  or  agency.  In  turn, 
the  lead  service  or  agency  developed  a  project  management  plan  that 
specifically  outlined  how  the  project  would  be  executed.  These  two 
efforts  established  the  “contract”  for  the  project  between  the  lead  agent 
and  the  Committee. 

The  projects  were  then  executed  by  the  lead  service  or  agency.  The 
Committee  established  program  corporate  investment  priorities  with  the 
goal  of  preventing  unnecessary  duplication,  encouraging  multiservice 
use  of  equipment,  and  providing  critically  needed  test  capabilities.  Pri¬ 
orities  were  to  be  determined  in  all  phases  of  program  management 
(planning,  programming,  budgeting,  and  budget  execution)  by  the  Com¬ 
mittee.  As  a  result,  the  Director,  by  assuming  m^jor  responsibilities  for 
the  Committee  that  managed  development  test  resource  funding,  placed 
his  independence  at  risk. 


dod’s  response  to  a  draft  of  this  report  stated  that  the  Test  and  Evalua¬ 
tion  Committee  under  the  Defense  Acquisition  Board  has  been  disestab¬ 
lished.  However,  we  were  told  that  a  dod  Test  and  Evaluation  Resource 
Council  may  be  created  and  may  be  co-chaired  by  the  Director.  At  pre¬ 
sent,  it  is  unclear  how  such  a  council  will  operate.  If  the  council  is  estab¬ 
lished  to  plan,  program,  and  budget  for  development  test  resources  and 
the  Director  serves  as  a  co-chairman  or  even  as  a  council  member,  we 


BThe  funding  for  the  program  is  found  in  the  appropriation  for  the  Deputy  Director,  Defense 
Research  and  Engineering  (Test  and  Evaluation),  Office  of  the  Secretary  of  Defense,  who  is  respon¬ 
sible  for  development  testing.  Fiscal  oversight,  distinct  from  the  Committee’s  responsibility  for  man¬ 
agement  and  corporate  priority  setting,  is  the  Job  of  the  Deputy  Director. 
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believe  the  Director  would  continue  to  exceed  the  role  of  providing 
advice  to  officials  responsible  for  development  test  matters. 

In  the  past,  the  Test  and  Evaluation  Committee  membership  consisted  of 
the  Committee  chair,  vice  chair,  members,  and  invited  participants/ 
advisers.  The  Committee  Chair  and  members  were  to  reach  a  consensus 
on  various  issues  brought  before  the  Committee.  The  invited  partici¬ 
pants  and  advisers  were  to  provide  information  on  specific  matters.  In 
keeping  with  dot&e’s  responsibilities  for  or&E  oversight  as  addressed  in 
its  statute,  it  would  be  more  appropriate  for  the  Director  to  be  an 
adviser  on  any  future  test  resources  council  that  is  created.  As  such,  the 
Director  should  continue  to  perform  oversight  and  policy  functions  by 
reviewing  test  plans  and  results  and  making  recommendations  to  the 
Secretary  of  Defense  so  that  test  resource  needs  are  given  appropriate 
consideration. 


Use  of  Early 
Operational 
Assessments  Is  Within 
DCT&E’s  Mission 


Our  review  showed  that  DOT&E’s  use  of  early  operational  assessments, 
which  dod  began  using  in  1988,  are  not  prohibited  by  dot&e’s  statute. 
They  allow  dot&e  to  offer  information  to  dod  acquisition  decisionmakers 
as  to  whether  weapon  systems  are  ready  for  or&E  and  whether  opera¬ 
tional  shortfalls  are  being  identified  and  corrected  early  in  the  acquisi¬ 
tion  process.  These  assessments,  which  are  based  primarily  on 
development  rather  than  or&E  data,  are  being  done  during  the  early 
phases  of  the  acquisition  process  when  actual  ct&e  results  are  not  avail¬ 
able.  However,  as  noted  in  a  previous  report,6  they  are  not  substitutes 
for  or&E  and  the  basis  for  the  assessments  should  be  fully  disclosed 
when  they  are  reported  to  congressional  and  dod  decisionmakers. 


According  to  the  Director,  an  independent  evaluation  is  made  of  the 
operational  assessments  developed  by  the  operational  test  agencies.  This 
evaluation  is  provided  as  advice  to  the  Under  Secretary  of  Defense  for 
Acquisition  through  the  Defense  Acquisition  Board  process,  which  is 
used  to  oversee  major  system  acquisitions. 


The  Director’s  Past 
Attempts  to  Redefine  Test 
and  Evaluation 


In  the  past,  the  Director  has  attempted  to  redefine  the  way  test  and 
evaluation  is  viewed  within  dod.  Until  the  Director  began  advocating  the 
use  of  early  operational  assessments,  initial  or&E  was  that  portion  of 
actual  or&E  done  throughout  the  acquisition  process  before  the  decision 
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to  proceed  to  production.  It  was  accomplished  using  a  prototype, 
preproduction  article,  or  a  low-rate  initial  production  article  as  the  test 
item.  The  “final  exam,”  or  the  latter  phase  of  initial  ot&e  usually 
entailed  dedicated  operational  testing  of  production  representative  test 
articles  using  typical  operational  personnel  in  as  realistic  a  combat  envi¬ 
ronment  as  possible. 

In  a  report  to  the  House  Armed  Services  Committee  dated  September  25, 
1987,  the  Secretary  of  Defense  proposed  to  redefine  initial  ot&e  as  that 
component  of  testing  initiated  at  program  inception,  to  forecast  opera¬ 
tional  effectiveness  and  suitability.  It  was  to  be  a  tool  to  provide  insights 
about  the  potential  operational  worth  of  a  system  throughout  its  acqui¬ 
sition  life.  As  opposed  to  doing  actual  ot&e,  initial  ot&e  would  take 
advantage  of  any  test  results  and  could  use  modeling,  simulation,  and 
paper  analyses  to  make  assessments.7 

dod’s  current  policy  requires  early  and  progressive  assessments  of  oper¬ 
ational  capability,  including  realistic  operational  testing  before  full-scale 
production  starts.  It  is  the  current  dod  perspective  that  an  operational 
test  agency  should  evaluate  all  pertinent  information  as  it  becomes 
available,  regardless  of  the  source,  for  input  into  early  periodic  opera¬ 
tional  assessments. 

On  January  26, 1990,  dod  issued  guidelines  that  prescribe  uniform  pro¬ 
cedures  for  preparing  and  approving  Test  and  Evaluation  Master  Plans. 
The  plans  describe,  among  other  things,  the  development  and  opera¬ 
tional  testing  to  be  performed  on  a  weapon  system  throughout  the 
acquisition  process.  These  guidelines  state  that  the  plans  will  show  how 
operational  assessments  and  testing  are  structured  at  each  acquisition 
decision  point.  Further,  they  show  how  operational  testing  will,  or  has, 
evaluated  the  system  in  an  environment  as  operationally  realistic  as 
possible.  If  ot&e  cannot  be  done  or  completed  early  in  the  development 
process,  then  this  is  to  be  clearly  stated  and  the  reason(s)  explained  in 
the  plans. 

The  guidelines  state  that  operational  assessments  are  done  before  or  in 
support  of  the  full-scale  engineering  development  phase.  They  are  used 
to  identify  significant  trends  noted  in  development  efforts,  program¬ 
matic  voids,  areas  of  risk,  adequacy  of  requirements,  and  the  ability  of 
the  program  to  support  adequate  operational  testing.  The  assessments 

According  to  a  DOT&E  official,  this  new  definition  for  initial  OT&E  was  synonymous  with  an  early 
operational  assessment 
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are  to  be  based  on  all  information  relevant  to  the  program,  and  can  be 

made  at  anytime.  However,  they  are  not  to  be  considered  substitutes  for 

ot&e  in  support  of  full-production  decisions.  < 


Early  Operational 
Assessments  Can  Be 
Useful 


Performing  early  operational  assessments  when  production  representa-  | 

tive  test  articles  are  not  available  is  a  step  forward  in  filling  a  void  in 
the  availability  of  actual  ot&e  results.  Furthermore,  the  assessments  are 
considered  advisory  in  nature  and  can  be  useful  to  decisionmakers.  For 
example,  the  C-17A  aircraft  was  the  first  major  acquisition  program  i 

that  relied  on  an  early  operational  assessment  as  a  decision-making 
tool.8  In  December  1987,  the  Deputy  Secretary  of  Defense  required  the  ! 

Air  Force  to  submit  an  assessment  of  the  C-17A  regarding  (1)  meeting 
low-rate  initial  production  criteria  and  (2)  progress  towards  initial  oper¬ 
ational  capability. 

In  September  1988,  the  Air  Force  Operational  Test  and  Evaluation 
Center  completed  the  C-l  7A  assessment  to  support  a  low-rate  initial  I 

production  decision  consisting  of  40  C-17A  aircraft.  Since  no  production 
representative  systems/subsystems  were  available,  no  operational  tests 
were  done.  The  tenter’s  assessment  was  based  on  all  available  informa¬ 
tion  sources  to  address  such  areas  as  attainment  of  the  production  cri¬ 
teria  listed  in  the  C-17A’s  Test  and  Evaluation  Master  Plan.  These 
sources  included,  but  were  not  limited  to,  results  of  critical  design 
reviews,  development  test  results,  demonstrations,  the  center’s  partici¬ 
pation  in  planning  committees,  program  documents,  and  interviews. 

In  January  1989,  the  Deputy  Secretary  of  Defense  approved  low-rate 
initial  production  of  the  C-17A  aircraft,  which  reduced  the  requested  l 

procurement  of  40  aircraft  to  10  aircraft.  Because  C-17A  aircraft  were  ' 

not  available  for  ot&e,  the  early  operational  assessment  was  beneficial  f 

in  that  it  provided  timely  insight  into  potential  problems  for  the  C-17A 
aircraft  based  on  essentially  development  test  data. 


Our  Earlier  Report  Dealt 
With  Disclosure  of  Early 
Operational  Assessments 


In  our  1989  report  to  the  Chairman,  Subcommittee  on  Seapower  and 
Strategic  and  Critical  Materials,  House  Committee  on  Armed  Services, 
we  evaluated  the  Navy’s  ot&e  done  before  decisions  were  made  on  the 
full-scale  development  or  low-rate  initial  production  of  weapon  systems. 
During  fiscal  years  1985  to  1988,  the  Navy  typically  approved  weapon 


sDOD  is  Just  beginning  to  use  assessments  in  the  acquisition  process.  As  a  result,  there  are  very  few 
examples  of  early  operational  assessments. 
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systems  for  full-scale  development  and,  in  many  cases,  for  low-rate  ini¬ 
tial  production  before  any  ot&e  was  completed.  However,  the  Navy  did 
do  a  limited  number  of  operational  assessments  that  supported  the  early 
milestone  decisions. 

Based  on  our  evaluation  of  dod’s  testing  policy  and  the  Navy’s  use  of 
operational  assessments,  we  recommended  in  our  1989  report  that  the 
Secretary  of  Defense  clarify  when  it  is  appropriate  for  decisionmakers 
to  rely  on  operational  assessments  that  do  not  include  the  operational 
testing  of  any  hardware  and  when  actual  ot&e  must  occur.  In  addressing 
this  issue,  the  Secretary  was  to  assure  the  independence  of  the  services’ 
ot&e  agencies  was  not  compromised  and  that  the  basis  of  the  weapon 
system  assessments  were  fully  disclosed  when  the  assessments  were 
reported  to  congressional  and  dod  decisionmakers. 

In  response  to  our  report,  dod  stated  that  it  agreed  with  our  recommen¬ 
dations.  dod  further  stated  that  the  policy  as  to  when  it  is  appropriate 
for  decisionmakers  to  rely  on  operational  assessments  would  be  included 
in  the  publication  of  dod  Manual  5000.3-M-5,  volume  3.  Due  to  changes 
in  dod’s  testing  policy  being  considered  as  part  of  dod’s  Defense  Manage¬ 
ment  Review,  many  directives  and  regulations  are  being  consolidated, 
revised,  or  terminated.  For  example,  dod  plans  to  publish  a  new  Direc¬ 
tive  5000. 19  addressing  the  acquisition  process  as  well  as  a  manual 
addressing  acquisition  documentation  and  reporting  procedures.  As  a 
result,  test  and  evaluation  policies  and  procedures  will  not  be  published 
until  sometime  in  the  future. 


Recommendation 


We  recommend  that  the  Secretary  of  Defense  keep  development  and 
operational  testing  independent  and  separate  by  ensuring  that  the 
Director  provides  advice  only  to  those  committees  or  councils  that  are 
responsible  for  the  planning,  programming,  and  budgeting  of  develop¬ 
ment  test  resources.  The  Director  should  perform  oversight  and  policy 
functions  by  reviewing  test  plans  and  results  and  making  recommenda¬ 
tions  to  the  Secretary  of  Defense  so  that  test  resource  needs  are  given 
appropriate  consideration. 


Agency  Comments  and 
Our  Evaluation 


dod  agreed  that  development  and  operational  testing  should  be  kept 
independent  and  separate.  However,  dod  disagreed  that  dot&e  had  man¬ 
aged  development  test  resources,  dod  stated  that  neither  the  Test  and 


9In  the  past,  DOD  Directive  5000.1  addressed  the  subject  of  mqjor  weapon  system  acquisitions. 
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Evaluation  Committee  nor  the  chairman  ever  managed  development  test 
resources  or  any  program  such  as  the  Central  Test  and  Evaluation 
Investment  Program  that  involves  such  resources.  As  such,  dod  has 
stated  it  was  already  in  compliance  with  our  recommendation  and  that 
the  Director  does  not  now  function  as  chairman  of  any  committee  or  any 
other  organization  responsible  for  managing  test  resources  nor  is  there 
any  intent  to  assign  such  responsibilities  to  the  Director. 

Although  we  recognize  that  test  and  evaluation  resource  management 
structures  are  presently  undergoing  changes,  the  Director  had  major 
responsibilities  chairing  a  committee  that  planned,  programmed,  and 
budgeted  for  development  test  resources.  As  Committee  Chairman,  the 
Director  exercised  mqjor  responsibilities  over  the  operations  of  the  Com¬ 
mittee.  Further,  the  Committee  was  responsible  for  planning,  program¬ 
ming,  and  budgeting  for  the  Central  Test  and  Evaluation  Investment 
Program.  Although  we  did  not  identify  examples  where  the  Director  had 
unduly  influenced  the  services,  we  believe  his  responsibilities  created 
the  perception  that  the  Director’s  independence  from  development  test 
matters  was  jeopardized  by  allowing  for  a  direct  influence  over  the  test 
assets  used  by  the  services. 

dod  told  us  that  a  corporate  mechanism  has  been  established  to  focus 
attention  on  the  need  for  test  resources,  dod  believes  that  this  attention 
has  forced  the  services  to  work  together  to  identify  requirements,  elimi¬ 
nate  duplication,  and  obtain  the  necessary  funds  to  support  the 
requirements. 

Based  on  dod’s  comments,  we  have  modified  our  recommendation  to 
better  emphasize  the  actions  we  believe  are  needed,  (dod’s  detailed  com¬ 
ments  and  our  evaluation  of  these  comments  are  presented  in  app.  II.) 


We  are  sending  copies  of  this  report  to  the  Secretaries  of  Defense,  the 
Navy,  Army,  and  Air  Force  and  to  interested  parties.  Copies  will  also  be 
made  available  to  others  on  request. 
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Please  contact  me  at  (202)  275-8400  if  you  or  your  staff  have  any  ques¬ 
tions  concerning  this  report.  Major  contributors  to  this  report  are  listed 
in  appendix  III. 


Sincerely  yours, 


Paul  F.  Math 

Director,  Research,  Development, 
Acquisition,  and  Procurement  Issues 
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Scope  and  Methodology 


To  assess  the  Director’s  role  as  Chairman  of  the  Test  and  Evaluation 
Committee,  we  reviewed  the  scope  of  dot&e’s  legislation  as  it  pertained 
to  development  test  matters.  In  particular,  we  focused  on  a  legislative 
provision  that  stated  that  the  Director  was  not  to  be  assigned  any 
responsibility  for  development  test  and  evaluation,  other  than  the  provi¬ 
sion  of  advice  to  officials  responsible  for  such  testing. 

We  interviewed  Office  of  the  Secretary  of  Defense  officials  and  reviewed 
documents  pertaining  to  the  Test  and  Evaluation  Committee.  We 
reviewed  the  Director’s  involvement  in  test  resource  sponsorship  as  the 
Committee  Chairman  and  in  the  normal  role  as  dct&e.  In  particular,  we 
interviewed  Office  of  the  Secretary  of  Defense  officials  and  reviewed 
documents  pertaining  to  the  Central  Test  and  Evaluation  Investment 
Program  because  the  program  is  used  to  acquire  primarily  development 
test  resources. 

To  assess  the  Director’s  use  of  early  operational  assessments,  we  evalu¬ 
ated  the  scope  of  dot&e’s  legislation  as  it  pertained  to  development  test 
matters  and  whether  that  office  was  responsible  for  development  test 
and  evaluation,  other  than  the  provision  of  advice  to  officials  respon¬ 
sible  for  such  testing. 

Since  the  early  operational  assessment  is  an  evolving  concept,  only  lim¬ 
ited  information  is  available  on  it.  Primarily,  we  reviewed  documents 
that  set  out  dct&e’s  policy  on  early  operational  assessments.  We  also 
interviewed  service  officials  from  the  Navy  Commander,  Operational 
Test  and  Evaluation  Force,  Norfolk,  Virginia;  Air  Force  Operational  Test 
and  Evaluation  Center,  Albuquerque,  New  Mexico;  and  the  Army  Opera¬ 
tional  Test  and  Evaluation  Agency,  Alexandria,  Virginia,  to  obtain  their 
views  on  early  operational  assessments. 

Our  review  was  performed  in  accordance  with  generally  accepted  gov¬ 
ernment  auditing  standards  between  October  1988  and  January  1990. 
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Now  on  pp.  1-2 


GAO  DRAFT  REPORT  -  DATED  APRIL  13,  1990 
GAO  CODE  396224  -  OSD  Case  8303 

"DOD  TESTING:  QUESTIONABLE  TASKS  PERFORMED  BY  THE  DIRECTOR, 
OPERATIONAL  TEST  AND  EVALUATION” 

DEPARTMENT  OF  DEFENSE  COMtENTS 

***** 

FINDINGS 

o  FINDING  A:  Background:  Office  of  the  Director,  Operational 
Test  and  Evaluation.  The  GAO  observed  that  the  Congress, 
concerned  that  operational  test  and  evaluation  was  not 
receiving  sufficient  emphasis  and  independent  oversight, 
established  the  Director,  Operational  Test  and  Evaluation 
(P.L.  98-94,  September  24,  1983)  .  According  to  the  GAO,  the 
statute  designated  the  Director  as  the  principal  operational 
test  and  evaluation  official  within  senior  DoD  management  to 
ensure  that  systems  being  acquired  are  ready  for  production. 
The  GAO  pointed  out  that,  to  ensure  the  independence  of  the 
Director,  Operational  Test  and  Evaluation,  on  several 
occasions,  the  Congress  legislated  the  organizational 
separation  between  development  and  operational  testing — 
the  two  primary  types  of  DoD  testing  during  the  acquisition 
process.  The  GAO  observed  that  the  Director,  Operational  Test 
and  Evaluation,  may  not  be  assigned  any  responsibility  for 
development  test  and  evaluation — except  to  provide  advice  to 
officials  responsible  for  such  testing.  (pp.  2-4/GAO  Draft 
Report) 

POP  RESEflMSE:  Concur. 


o  FINDING  3:  The  Director  Should  Mr»t  Manage  Test  Resources. 
The  GAO  found  that  the  Director,  as  Chairman  of  the  Test  and 
Evaluation  Committee,  had  recently  become  involved  in  the 
management  of  development  test  resources.  The  GAO  explained 
that  managing  development  test  resources  goes  beyond  merely 
providing  advice  to  officials  responsible  for  development  test 
matters  and  puts  at  risk  the  independence  of  the  Director  from 
development  test  matters  because  it  gives  the  Director  control 
and  influence  over  the  types  of  development  test  assets  used 
by  the  Services. 

The  GAO  pointed  out  that  in  1987,  the  Congress  inserted 
language  in  the  National  Defense  Authorization  Act  for  Fiscal 
years  1988-89,  relating  to  the  Office  of  the  Director, 
Operational  Test  and  Evaluation,  which  states  "The  Director 

Enclosure 
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may  not  be  assigned  any  responsibility  for  developmental  test 
and  evaluation,  other  than  the  provision  of  advice  to 
officials  responsible  for  such  testing."  According  to  the 
GAO,  the  conference  report  on  this  Act  had  the  language 
inserted  to  ensure  that  responsibilities  for  operational 
testing  are  separate  from  functions  associated  with 
development  testing.  The  GAO  noted  that  the  conference  report 
further  stated  that  the  Secretary  of  Defense  should  refrain 
from  any  realignment  or  new  arrangement  of  test  and  evaluation 
activities,  oversight  responsibilities,  or  functions. 

The  GAO  further  referred  to  a  recently  issued  GAO  report,  TEST 
AND  EVALUATION :  An  Assessment  of  DoD's  Operational  Test  and 
Evaluation  Capability  Improvement  Program,  dated  March  20, 
1990  (OSD  Case  8137) ,  which  criticized  the  management  of  the 
Operational  Test  and  Evaluation  Capability  Improvement 
Program.  The  GAO  concluded  that  the  Director,  Operational 
Test  and  Evaluation,  was  performing  management  functions  by 
acquiring  test  resources  used  in  testing  major  weapon  systems . 
The  GAO  asserted  that  management  of  test  resources  dilutes  the 
Director's  oversight  function  because  it  gives  the  Director, 
Operational  Teat  and  Evaluation,  a  direct  influence  over  the 
types  of  test  assets  acquired  and  used  by  the  Services.  The 
GAO  concluded  that  the  Director  may  be  reluctant  to  criticize 
a  test  plan  that  contains  inadequate  testing  resources  if  his 
office  was  responsible  for  managing  or  played  a  part  in  the 
Now  on  pp.  3-4  acquisition  of  such  resources.  (pp.  1-2,  pp.  4-5/GAO  Draft 

Report) 


DoD  RESPONSE;  Nonconcur.  The  Department  acknowledges  that 
there  were  some  statements  in  the  early  correspondence 
creating  the  Central  Test  and  Evaluation  Investment  Program 
that  assigned  management  responsibility  to  the  Test  ?nd 
Evaluation  Committee  (hereafter  referred  to  as  the  Committee) 
and  the  Chairman  of  the  Committee  (hereafter  referred  to  as 
See  comment  1.  the  Chairman).  In  practice,  however,  the  Chairman  did  not 

manage.  As  specified  in  the  Committee  charter,  the  Committee 
provided  a  forum  and  catalyst  for  the  review  of  DoD  test 
resource  matters  and  recommended  alternatives.  It  did  not 
have  management  authority  under  the  charter.  Moreover,  the 
Chairman  did  not  have  decision  authority;  consequently,  be 
could  not  manege  test  resources.  The  Chairman's  role  was  to 
facilitate  the  review  process  and  cause  agreement  to  occur 
where  that  was  possible.  In  that  capacity,  for  all  members 
of  the  Committee,  whether  they  were  judged  to  be  of  the 
development  or  operational  test  communities,  the  Director, 
Operational  Test  and  Evaluation  or  his  representative  provided 
"advice  to  officials  responsible  for  such  testing." 
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Committee  was  established  to  support  the  Under  Secretary  of 
Defense  for  Acquisition  who,  by  law,  is  responsible  for 
acquiring  DoD  weapon  systems  and  the  development  testing  of 
such  systems .  The  GAO  noted  that  the  Committee  was  to  provide 
a  forum  for  hey  representatives  from  the  Office  of  the 
Secretary  of  Defense,  the  Services,  and  other  agencies  to 
identify  and  resolve  issues  and  to  develop  recommendations  on 
the  DoD  acquisition  system  in  the  test  and  evaluation  area. 
They  learned  that  the  Deputy  Secretary  of  Defense  appointed 
the  Director,  Operational  Test  and  Evaluation  as  the  Chairman 
of  the  Committee. 

The  GAO  explained  that,  as  the  Chairman,  the  Director 
initially  provided  advice  and  presented  recommendations  on 
various  test  and  evaluation  issues  to  the  Deputy  Secretary  of 
Defense  for  a  decision.  The  GAO  found,  however,  that  in 
1986,  the  issues  included  such  things  as  a  proposed 
recommendation  to  improve  space  system  test  capabilities.  The 
GAO  further  found  that,  in  1988,  the  Director  recommended 
establishing  the  Central  Test  and  Evaluation  Investment 
Program,  which  received  $83  million  in  FY  1990  for  the  purpose 
of  acquiring  primarily  development  test  resources  for  new 
weapon  systems.  The  GAO  acknowledged  that  the  Director 
believed  that  the  lack  of  development  and  operational  test 
investments  by  the  Services  was  reaching  crisis  proportions. 

The  GAO  concluded  that,  instead  of  continuing  to  serve  in  a 
advisory  capacity  by  presenting  recommendations  on  test 
resources  to  the  Deputy  Secretary  of  Defense,  the  Director, 
Operational  Test  and  Evaluation,  became  a  manager  of  those 
resources  by  taking  on  the  responsibility  for  the  program.  The 
GAO  emphasized  that,  in  November  1988,  the  Deputy  Secretary 
of  Defense  created  the  program  and  directed  that  it  be  managed 
by  the  Committee.  The  GAO  concluded,  therefore,  that  the 
Committee  Chairman  became  ultimately  responsible  and 
accountable  for  the  management  and  funding  of  development  test 
resources . 

While  agreeing  the  program  is  executed  by  the  Military 
Services,  the  GAO  further  concluded  that  the  Committee  sets 
program  corporate  investment  'priorities  with  the  goal  of 
preventing  unnecessary  duplication,  encouraging  multi-Service 
use  of  equipment,  and  providing  critically  needed  test 
capabilities.  The  GAO  observed  that  the  priorities  are  to  be 
determined  in  all  phases  of  program  management  by  the 
Committee;  therefore,  the  Director,  Operational  Test  and 
Evaluation,  by  controlling  development  test  resource  funding. 
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Now  on  pp.  4-5 


See  comment  3. 


See  comment  1 . 


See  comment  4. 


Now  on  pp.  5-6 


has  exceeded  the  mission  of  providing  advice  to 
decisionmakers.  (pp.  1-2,  pp.  6-7/GAO  Draft  Report) 


DoD  RESPONSE ;  Partially  concur.  The  majority  of  the  test 
resources  that  will  be  acquired  under  the  Central  Test  and 
Evaluation  Investment  Program  will  directly  support  both 
operational  and  developmental  testing.  Those  test  resources 
that  do  not  directly  support  operational  testing  will  provide 
data  that  will  be  used  to  support  operational  effectiveness 
and  suitability  assessments  or  they  will  be  used  as 
evaluation/analysis  tools. 

The  Department  acknowledges  that  a  Deputy  Secretary  Memorandum 
did  direct  the  Committee  to  manage  the  Central  Test  and 
Evaluation  Investment  Program  and  that  a  subsequent  Committee 
Memorandum  stated  that  the  "...Chairman  ...  is  ultimately 
responsible  and  accountable  for  the  management  and  effective 
allocation  of  [Central  Test  and  Evaluation  Investment  Program] 
funds."  However,  in  recognition  of  the  concerns  of  the 
Congress,  in  actual  practice,  the  Director  never  exercised  his 
Department  mandate  to  "manage"  the  Central  Test  and  Evaluation 
Investment  Program.  As  defined  above,  the  Director  has  never 
managed  the  Central  Test  and  Evaluation  Investment  Program. 

Rather  than  managing,  as  Chairman  of  the  Committee,  the 
Director  ensured  a  systematic  process  for  review  of  all  test 
resource  matters.  In  fact,  in  recommending  the  management 
approach  for  the  Central  Test  and  Evaluation  Investment 
Program,  the  Director  proposed  that  the  funding  be  placed  in 
the  Deputy  Director  Defense  Research  and  Engineering  (Test  and 
Evaluation)  appropriation,  recognizing  that  office  as  the 
Department  test  resource  manager. 


o  FINDING  D:  Ttin  rn—irt-.ae' ■  Future  ia  Uncertain.  The  GAO 
explained  that  the  continued  need  for  the  Committee  is  being 
questioned  as  a  part  of  the  DoD  Defense  Management  Review. 
According  to  the  GAO,  the  Committee  functions  may  be  placed 
under  a  council  that  could  possibly  be  chaired  by  the 
Director,  Operational  Test  and  Evaluation.  The  GAO  observed 
that  it  is  currently  unclear  whether  such  a  council  would 
continue  to  manage  development  test  resources.  The  GAO 
concluded,  however,  that  if  the  council  does  manage  such 
resources  and  the  Director  is  the  council  Chairman,  then  the 
Director,  Operational  Test  and  Evaluation,  would  continue  to 
be  in  the  position  of  managing  development  test 
resources — thus  still  going  beyond  the  mission  of  only 
providing  advice  to  officials  responsible  for  development  test 
matters.  (pp.  1-2,  p.  8/GAO  Draft  Report) 


Pafe  19 


QAO/NSLAD40-138  DOD  Tastia* 


Appendix  n 

Comments  From  the  Department  of  Defenae 


Pod  RESPONSE ;  Partially  Concur.  The  Test  and  Evaluation 

Committee  under  the  Defense  Acquisition  Board  has  been 
disestablished.  However,  the  DoD  Test  and  Evaluation  Activity 
Consolidation  Study,  conducted  as  a  part  of  the  Defense 
See  comment  2.  Management  Review,  does  propose  the  establishment  of  a  DoD 

Test  and  Evaluation  Resource  Council,  independently  chartered 
under  the  Deputy  Secretary  of  Defense.  If  the  Deputy 
Secretary  approves  the  Council,  it  will  be  chartered  in  such 
a  way  that  there  is  neither  the  perception  or  fact  of 
organizational  bias. 


o  FINDING  E:  Dae  of  Early  Operational  Assessments  la  Within  the 
DOTtE* a  Mission.  The  GAO  found  that  the  use  of  early 
operational  assessments,  which  the  DoD  began  using  in  1988, 
are  not  prohibited  by  the  Director,  Operational  Test  and 
Evaluation  statute.  The  GAO  explained  that  the  assessments 
allow  the  Director  to  offer  information  to  DoD  acquisition 
decisionmakers  as  to  whether  weapon  systems  are  ready  for 
operational  test  and  evaluation  and  whether  operational 
shortfalls  are  being  identified  and  corrected  early  in 
the  acquisition  process.  The  GAO  pointed  out  that  these 
assessments,  which  are  based  primarily  on  development  rather 
than  operational  test  and  evaluation  data,  are  being  done 
during  the  early  phases  of  the  acquisition  process,  when 
actual  operational  test  and  evaluation  results  are  not 
available.  The  GAO  emphasized,  however,  that  they  are  not 
substitutes  for  operational  test  and  evaluation  and  the  basis 
for  the  assessments  should  be  fully  disclosed  when  they  are 
reported  to  congressional  and  DoD  decisionmakers,  as  noted  in 
its  prior  report,  NAVY  WEAPONS  TESTING;  Defense  Policy  on 
Early  Operational  Testing."  dated  May  8,  1989  (OSD  Case  7800)  . 

According  to  the  GAO,  the  Director  stated  that  an 
independent  evaluation  is  made  of  the  operational  assessments 
developed  by  the  operational  test  agencies.  The  GAO  noted 
that  this  evaluation  is  provided  as  advice  to  the  Under 
Secretary  of  Defense  for  Acquisition  through  the  Defense 
Acquisition  Board  process  that  is  used  to  oversee  major  system 
Now  on  p.  6  acquisitions.  (pp.  1-2,  pp.  8-14/GAO  Draft  Report) 


DoD  RESPONSE :  Concur. 
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Now  on  p.  9 


See  comments  1  and  2. 


RECOMMENDATION 


RECOMMENDATION ;  The  GAO  recommended  that  the  Secretary 
of  Defense  assure  that  development  and  operational  testing  be 
kept  independent  and  separate.  (The  GAO  emphasized  that  the 
Secretary  should  ensure  that  the  Director  does  not  chair  or 
manage  committees  or  programs  that  involve  development  test 
resources  that  can  compromise  the  independence  of  development 
and  operational  testing.)  (p.  15/GAO  Draft  Report) 

PoP  RESEOHSE;  Concur.  The  Department  is  already  complying 
with  this  recommendation  because  (1)  the  Director  does  not 
manage  any  test  resources  and  (2)  there  is  no  intent  to  assign 
any  such  responsibilities  to  him.  The  Secretary  has  found  the 
Director' s  counsel  on  Operational  Test  and  Evaluation-related 
budgetary  and  financial  matters  to  be  invaluable  and  intends 
to  ensure  that  such  advice  remains  available  through 
established  Department  corporate  and  functional  mechanisms. 
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The  following  are  gao’s  comments  on  dod’s  letter  dated  June  6, 1990. 


GAO  Comments 


1.  As  explained  in  the  report,  the  Director  had  major  responsibilities 
chairing  a  Committee  that  planned,  programmed,  and  budgeted  for 
development  test  resources.  Instead  of  continuing  to  serve  in  an  advi¬ 
sory  capacity  by  presenting  recommendations  on  test  resources,  the 
Director  took  on  responsibilities  for  managing  these  resources.  This  type 
of  participation  placed  at  risk  the  Director’s  independence  from  develop¬ 
ment  test  matters  because  it  gave  the  Director  influence  over  the  types 
of  test  assets  acquired  and  used  by  the  services. 

We  modified  our  report  to  discuss  in  greater  detail  how  we  believe  the 
Director  influenced  the  Committee  in  development  test  resources.  We 
explain  that  the  Chairman  had  major  roles  and  responsibilities 
regarding  the  operations  of  the  Committee.  (See  p.  4.)  In  addition,  the 
Committee’s  resource  panel  served  as  the  working  arm  for  the  Com¬ 
mittee.  For  example,  the  resource  panel  developed  a  “contract”  between 
the  Committee  and  lead  service  or  agency  on  how  specific  projects 
would  be  executed.  (See  p.  5.) 

2.  While  we  recognize  that  the  Committee  has  been  disestablished,  we 
have  included  information  in  the  report  about  it  because  the  Committee 
offers  a  historical  perspective  on  how  such  a  council  could  function.  For 
example,  the  Committee’s  charter  provides  a  framework  as  to  how  the 
dod  Test  and  Evaluation  Resource  Council's  charter  could  be  structured. 
By  reviewing  the  Committee’s  charter,  we  found  that  the  Chairman  had 
major  responsibilities  dealing  with  the  operations  of  the  Committee  that 
would  not  be  appropriate.  As  such,  the  Director  had  influence  over  the 
types  of  test  assets  used  by  the  services.  (See  p.  4.) 

dod  states  that  the  Council  will  be  chartered  in  such  a  way  that  there  is 
neither  the  perception  or  fact  of  organizational  bias.  In  dod’s  official  oral 
comments,  we  were  told  that  the  Council  could  possibly  be  co-chaired  by 
the  Director.  We  have  modified  the  report  to  show  the  past  membership 
of  the  Test  and  Evaluation  Committee.  (See  p.  6.)  By  reviewing  the 
charter  of  the  Committee,  we  found  that  the  membership  consisted  of 
the  Committee  chair,  vice  chair,  members,  and  invited  participants/ 
advisers.  The  invited  participants  and  advisers  were  to  participate 
based  on  specific  matters  to  be  addressed.  In  keeping  with  the  Director’s 
responsibilities  for  ot&e  oversight  as  addressed  in  its  statute,  we  believe 
the  Director  should  more  appropriately  be  an  adviser  on  any  future 
council  that  may  be  created. 
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3.  We  believe  that  the  Director  should  not  have  provided  more  than 
advice  to  officials  responsible  for  development  testing  and  continue  to 
believe  that  the  Director  should  not  exceed  providing  advice  by 
becoming  involved  with  development  test  resources  or  any  combination 
of  development  and  ot&e  test  resources.  In  addition,  we  recently  issued  a 
report  that  criticized  dot&e’s  management  of  or&E  resources.  We  recom¬ 
mended  that  the  Secretary  of  Defense  direct  and  assure  that  the  services 
plan,  program,  and  budget  for  adequate  test  resources  needed  to  con¬ 
duct  operational  testing  of  weapon  systems  effectively.  In  April  1990, 
the  Director  emphasized  to  the  services  the  importance  of  their  plan¬ 
ning,  budgeting,  and  programming  for  adequate  test  resources. 

4.  We  agree  that  the  Director  proposed  that  the  funding  for  the  program 
be  placed  in  the  Deputy  Director,  Defense  Research  and  Engineering 
(Test  and  Evaluation)  appropriation.  However,  the  Deputy  Director  was 
only  to  be  the  fiscal  agent  for  the  funds.  The  Committee,  on  the  other 
hand,  was  responsible  for  management  and  corporate  priority  setting. 
Priorities  were  to  be  determined  in  all  phases  of  program  management 
(planning,  programming,  budgeting,  and  budget  execution).  In  effect,  we 
believe  this  constituted  control  over  development  test  resource  funds. 
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